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RESEARCH  TITLE 


An  Investigation  of  One  and  Three  Paiameter  Item  Response 
Models  in  a  Field  Setting,  with  Implications  for  Computerized 
Adaptive  Testing 


Dr.  Bronel  R.  Whelchel,  PhD 
Morgan  State  University 
Baltimore,  Maryland 

Principal  Investigator 


"The  views  and  conclusions  conxained  in  this  document  are 
those  of  the  authors  and  should  not  be  interpreted  as  neces¬ 
sarily  representing  the  official  policies  or  endorsements, 
either  expressed  or  implied,  of  the  Air  Force  of  Scientific 
Research  or  the  US  Government." 


The  proposed  efforts  were  to  compare  one  and  three  parameter 
models  in  a  student  population.  The  student  population  were 
to  be  Black  ROTC  Students  presently  enrolled  at  Morgan  State 
University,  Baltimore,  Maryland.  A  computer  testing  environment 
was  to  eventually  emerge  from  this  study  and  from  a  Military 
standpoint  this  would  reduce  test  taking  time  25%,  therefore 
reducing  the  cost  of  such  efforts  by  30%. 

The  following  things  were  accomplished: 

One  trip  to  Human  Resources  Laboratory,  Brooks  Air  Force 
Base,  Texas  resulted  in  a  two  day  conference  with  the  authors 
of  the  test  in  order  to  ascertain  the  nature  and  scope  of 
the  test  material. 

Secondly,  a  three  track  computer  tape  was  purchased  so 
that  those  test  items  and  test  data  already  prepared  at  Brooks 
Air  Force  Base  in  the  Human  Resources  Lab  under  the  direction 
of  Drs.  Valentine  and  Ree  could  be  utilized  here  at  Morgan 
State  University.  Tlis  called  for  making  data  that  was  intended 
for  the  Univac  1108  compatible  with  Morgan  State  College 
Univac  9300.  The  turn  around  time  for  a  finished  product 
took  approximately  three  months. 


Air  Force  Office  of  Scientific  Research 
Air  Force  Systems  Command  USAF 
Attention:  Dr.  Haddad 
Bolling  4ir  Force  Base 
Washington,  DC  20332 


SUBJECT:  Scientific  Report  Mini  Grant  AFOSR-80-0191 


This  report  will  take  the  following  format: 

1.  Research  Title  and  Number 

2.  Description  of  the  Problem 

3.  Description  of  Results 

4.  Discussion  of  the  Results 

5.  Conclusions  in  form  of  a  Proposal  as  an  outgrowth 
of  this  mini  grant 
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RESEARCH  TITLE:  An  Investigation  of  One  and  Three  Parameter 

Item  Response  Models  in  a  Field  Setting, 
with  Implications  for  Computerized  Adaptive 
Testing 
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1.  OBJECTIVE  ADAPTIVE  TESTING 


"Adaptive"  testing  is  one  of  a  number  of  terms  used  to 
describe  a  procedure  whereby  the  test  items  that  comprise 
an  individual’s  test  are  selected  during  the  test  itself. 

Some  of  the  other  terms  used  interchangeably  with  adaptive 

testing  include  tailored  testing,  branched  testing,  programmed 

testing,  and  individualized  testing.  The  term  "adaptive" 

was  chosen  because  these  tests  adapt  themselves  to  the  examinee; 

different  persons  answer  different  items,  with  the  items 

chosen  sequentially  to  suit  the  individual  examinee's  performance. 

The  general  objective  of  computer  driver  adaptive  testing 
is  to  accurately  estimate  an  individual's  position  on  the 
underlying  trait  the  test  purports  to  measure.  Conventional 
paper-and-pencil  test  administration  typically  suffers  from 
several  sources  of  error  in  the  measurement  of  an  individual's 
ability.  Conventional  peaked  tests  are  designed  to  discriminate 
most  effectively  at  a  single  ability  level  and  thus  assume 
that  most  individuals  taking  the  test  fall  into  this  category. 

When  they  do  not,  the  accuracy  of  estimation  of  their  status 
on  the  trait  becomes  progressively  more  inaccurate  as  their 
ability  deviates  from  this  point. 

The  result  of  this  lower  precision  of  measurement  is 
lower  overall  reliability,  and  lower  validity  as  well. (Weiss, 

1974)  .  Lord  (1970,  1971a,  1971,  1971d,  1971e)  and  Hicic  (1951) 
have  concluded  that  a  test  score  most  accurately  reflects 
an  individual's  ability  when  the  probability  of  a^Correct 
response  to  an  item  is  .5  for  that  individual.'  Conventional 
tests  obviously  cannot  meet  the  requirements  of  .5  probability 
for  all  examinees  as  all  items  must  be  given  to  all  examinees. 

A  second  source  of  errors  results  from  not  extracting 
all  the  information  contained  in  an  examinee's  answers  to 
questions.  Conventional  tests  typically  use  number  of  pro¬ 
portion  of  items  discrimination. ^  Two  recent  developments 
have  enabled  the  psychometrician  to  more  accurately  assess 
the  status  of  individuals  on  measurable  traits:  adaptive 
testing  and  latent  trait  test  theory.  Adaptive  testing  enables 
adapting  each  test  to  fit  characteristics  of  each  individual 
tested.  ’-vThus,  it  allows  the  presentation  of  the  most  informative 
items  to  an  examinee  to  be  used  as  the  basis  of  that  individual's 
test  score.  Latent  trait  test  theory  allows  the  calculation 
of  ability  estimates  in  the  same  metric  for  each  examinee 
and  permits  comparisons  among  scores  even  though  each  may 
have  taken  a  different  subset  of  items  (Lord,  1970,  Wood,  1973). 


The  following  contractors  to  the  Office  of  Naval  Research 
are  engaged  in  adaptive  testing  studies:  The  Psychometric 
Methods  Program,  University  of  Minnesota;  the  Tailored  Testing 
Research  Laboratory,  University  of  Missouri;  Department  of 
Psychology,  University  of  Southern  California.  The  AFOSR 
and  AFHRL  have  adaptive  testing  contracts  with  Professors 
Ron  Hambleton  and  Hariharan  Swaminathan  at  the  University 
of  Massachusetts.  The  proposed  study  does  not  duplicate 
other  efforts,  but  rather  is  complimentary.  Results  of  the 
proposed  effort  will  advance  the  state  of  knowledge  in  this 
important  area. 

As  was  evident  from  paper  presented  at  the  1979  adaptive 
testing  conference  (CAT,  1979),  there  is  an  interest  in  real 
data  comparisons  of  one  and  three  parameter  item  response 
models.  It  is  possible  that  adaptive  testing  procedures 
might  have  a  high  utility  for  small  groups  when  the  one  parameter 
model  is  used. 

The  problem  involved  in  this  study  was  to  copy  onto  a 
computer  tape  those  test  items  developed  by  Drs.  Valentine 
and  Ree,  Human  Resources  Laboratory,  Brooks  Air  Force  Base, 
Texas.  These  items  were  prepared  for  a  Univac  1108,  Brooks 
Air  Force  Base  and  written  in  FORTRAN. 

The  tape  was  received  here  on  or  about  November  1980. 

Several  operations  had  to  take  place  before  the  Project  Director 
could  ascertain  how  to  prepare  the  tape  for  assemilination 
and  compatibility  with  our  system.  The  tape  was  finally 
converted  and  made  compatible  with  Morgan  State  University 
Univac  9300. 

The  tape  conversion  involved  the  following  computer 
personnel : 


1.  Mr.  Ronald  Gale,  IBM  specialist  on  loan  to  Morgan 
State  University  for  one  year. 

2.  Mr.  Charles  Rogers,  Human  Resources  Laboratory, 
Brooks  Air  Force  Base,  San  Antonio,  Texas. 

3.  Dr.  M.  Ree,  Human  Resources  Lab,  Brooks  Air  Force 
T’  Base,  Texas 

The  tape  is  now  marented,  converted  and  ready  for  Phase 
II  of  this  project. 

Phase  II  of  this  project  consists  of  a  major  proposal 
to  be  presented  to  interested  Armed  Forces  Research  Agencies 
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for  funding  beginning  the  school  /ear  1981-32.  A  copy  of 
this  proposal  is  hereby  submitted  for  your  examination. 
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